Synchronization of multiple media streams

ABSTRACT

A method of synchronizing digitized audio information and digitized picture information establishes a reference time base based on timecodes included with the audio and picture information. The reference time base may be used to synchronize output between a digitized source and one which may be digitized or undigitized.

This application is a continuation of application Ser. No. 08/713,107,filed Sep. 16, 1996 and U.S. Pat. No. 5,745,637 entitled SYNCHRONIZATIONOF DIGITAL VIDEO WITH DIGITAL AUDIO, which is a continuation ofapplication Ser. No. 08/344,409 filed Nov. 23, 1994, now U.S. Pat. No.5,557,423, issued Sep. 17, 1996, which is a continuation of applicationSer. No. 08/048,645, filed Apr. 16, 1993, abandoned.

BACKGROUND

1. Field of the Invention

The present invention relates generally to fields of motion picture,television and other multi-media production. More particularly, theinvention relates to methods for synchronizing the media comprising amotion picture, television or other multi media production so as tofacilitate viewing and editing.

2. Prior Art

Synchronization of multiple media sources is a problem which has plaguedthe motion picture production industry since the days of the earlytalkie movies. The techniques by which images are recorded on movie filmand the techniques by which sound is recorded on a suitable soundrecording medium naturally differ. Therefore, it has been common, sincethose early days, for two independent pieces of equipment to be used torecord image and sound.

In the early days, images were invariably recorded as a sequence offrames on a continuous strip of photographic film. In contrast, soundwas recorded as a microscopic wavy groove in either a wax cylinder or awax disk. Some audio was recorded on the set, but much audio wasrecorded while observing a screening of the film.

Synchronization of a projected picture with corresponding audio wasquite crude. Picture and audio would be started at a reference pointestablished at the program. The speed of one or the other would bemanually adjusted when loss of synchronization was observed. However,such techniques provide very poor synchronization, are unsuitable forsynchronizing long programs and are unsuitable for synchronizingmultiple tracks of audio with picture.

In modern studios, picture may be recorded on film or video tape, in avariety of formats. Audio is likely to be recorded on magnetic recordingtape, also in a variety of formats. Whenever multiple tracks of sourcematerial are used to create a final production or composition,synchronization of one or more pictures with one or more tracks of audiomay be required.

The use of a customary slate at the beginning of each take of a scenehelps facilitate synchronization of audio recorded on the set withpicture recorded on the set. The slate produces a sharp and distinctsound on the audio recording, simultaneous with an identifiable visualevent, such as a bar hitting the top of the board. Additionally, duringrecording, the camera and the audio recording equipment may bemechanically or electrically synchronized, so that corresponding lengthsof film and audio media are used. Consequently, to synchronize thesesource media for editing or viewing requires only that the sound of theslate in the audio track be aligned with the image of the bar hittingthe top of the board. Then the two media may be mechanically orelectrically locked together for playback or editing.

For greater flexibility, including the use of multiple tracks recordedeither on the set or off the set, timecodes have been developed foraiding in the establishment and maintenance of synchronization in thecircumstances such as described above. One type of commonly usedtimecode is that developed by the Society for Motion Picture andTelevision Engineers (SMPTE), which may be imprinted in an unused marginof the film or audio media during filming so as to indicate an absolutetime reference for each frame of film. The imprinting of the SMPTEtimecodes may be human readable, e.g., as numbers in the margin of thefilm, or may be machine readable, e.g. as bar codes in the margin of thefilm. Another time reference used on film media is the Eastman KodakKEYKODE number. Among other things, the KEYKODE number acts as a rolland frame counter, by which any frame of a production may be absolutelyidentified. It is also recorded in the film margin, but KEYKODES arerecorded on film stock at the time of manufacture.

Synchronization during playback may be achieved as follows. First, allthe media are aligned manually to a starting point. During playback,timecodes are read off all of the media to be synchronized. The playbackspeeds of the media are then adjusted so as to cause the timecodes readoff the media to coincide throughout the playback. Such systems employstandard feedback techniques to cause the playback speeds of all themedia but one to follow the playback speed of the one.

However, the techniques discussed above are inadequate for modernelectronic production and editing. As discussed above, currenttechniques still require physically aligning and synchronizing sourcemedia each time a source is to be played or edited. As noted above, thisprocedure requires manual alignment. It is therefore slow,labor-intensive and prone to error. Furthermore, at stages of productionbefore a final editing sequence has been created, it is difficult tosynchronize multi-track audio with cut film.

Systems are also available which can synchronously display digitizedaudio and video data. One such system is disclosed in Peters et al.,U.S. Pat. No. 5,045,940. However, such systems assume synchronizationbetween sources has previously been established. Another such system iscommonly known as a lock box. A lock box performs crude synchronizationbetween video and film. However, lock boxes do not accuratelysynchronize frame by frame, because they depend upon the frame rate ofthe source media, which differs between video and film.

Therefore, it is a general aim of the present invention to provide amethod for synchronizing picture and audio in modern production and postproduction facilities. More generally stated, it is an aim of thepresent invention to provide a method of synchronizing a multi-mediapresentation at all stages of production and post-production.

SUMMARY OF THE INVENTION

In accordance with one aspect of the present invention, there isprovided a method for synchronizing a stream of digitized picture datawith a stream of digitized audio data. The stream of digitized picturedata includes a stream of picture timecodes and the stream of audio dataincludes a stream of audio timecodes. The method includes steps of:establishing a referenced time base synchronous with one of the streamof picture time codes and the stream audio timecodes; associatinganother of the stream of picture time codes and the stream of audiotimecodes with the reference time base; and, storing as a track groupsynchronous over a range of time for which the picture data and theaudio data coincide, the digitized stream of picture data including thepicture timecodes, the digitizes stream of audio data including theaudio timecodes and the referenced timebase.

According to another aspect of the present invention, the method mayinclude steps of digitizing the stream of picture data, digitizing thestream of audio data, decoding the stream of picture timecodes anddecoding the stream of audio timecodes.

In accordance with another aspect of the present invention, there isprovided apparatus for synchronously displaying film dailies andcorresponding audio. Such apparatus may include a film projector, meansattached to the film projector for producing a stream of picture timecodes, means for storing digitized audio data having a referenced timebase, and means responsive to the stream of picture timecodes forreproducing the audio data synchronous with the picture timecodes.

In accordance with yet another aspect of the present invention, there isprovided a negative cutter apparatus. The negative cutter may include async block having a means for determining a starting frame number, ashaft and coder for giving a continuous indication of the position ofthe media and a digital playback system for comparing current mediaposition with cuts indicated on a digital workprint.

BRIEF DESCRIPTION OF THE DRAWING

In the drawing, wherein like reference numerals indicate like elements:

FIG. 1 is a block diagram illustrating digitization of multiple mediasources and establishment of a reference timebase for the digitizedsources;

FIG. 2 is a block diagram illustrating the elements of a system suitablefor accomplishing the digitization and synchronization of FIG. 1;

FIG. 3 is a block diagram illustrating one system for synchronousplayback of digitized source media in conjunction with the displaynon-digitized source media; and

FIG. 4 is a block diagram of a negative cutter system employing thesynchronization techniques of the present invention.

DETAILED DESCRIPTION

The invention will be better understood by reading the followingdescription of embodiments of the invention in conjunction with thedrawing.

The present invention seeks to establish synchronization betweenmultiple media sources which are being digitized for further productionwork. One context in which this may be desirable is that described inpending U.S. patent application Ser. No. 07/920,260, filed Jul. 27,1992, of Michael Wissner for A METHOD AND APPARATUS FOR REPRESENTING ANDEDITING MULTI-MEDIA COMPOSITIONS, incorporated herein by reference. Ofparticular interest is the description of the data structures containedtherein.

A method of achieving synchronization between multiple sources, inaccordance with one aspect of the present invention is now described inconnection with FIG. 1. Apparatus suitable for practicing this methodwill be described below in connection with FIG. 2.

This aspect of the present invention makes use of timecodes and timecodereaders known in the art and manufactured by AATON and ARRIFLEX undertheir respective trademarks. Source material, such as film 101 and audiotape 103 may recorded so as to include timecodes according to one of thecommon systems noted above. The timecodes are recorded periodically onthe medium, so as to establish an exact time relationship between eachelement of the medium and some absolute reference time. For example,each frame of a film 101 has a timecode associated therewith and printedon the edge of the film adjacent to the frame. The recorded timecoderepresents the absolute time at which that frame was recorded. Duringthe recording of audio tape 103, timecodes are incorporated whichcorrespond to the timecodes representing the frames of film 101. Thuswhen audio 103 is recorded at the same time as film 101 is shot, eachtime a timecode is recorded on film 101 the same timecode is recorded onaudio tape 103. However, the proper relationship may be established evenwhen audio is recorded separately from film. In that case, the absolutetime indicated by the recorded timecodes on audio tape 103 may differfrom those of film 101. Furthermore, if different frame rates were usedfor picture and audio, then the time between timecodes recorded on audiotape 103 will differ from the time between the timecodes recorded onfilm 101. However, since timecodes represent absolute time, it is stillpossible to identify any point on audio tape 103 as corresponding with aframe on film 101 by use of the timecodes.

In a subsequent processing step, both picture and audio may be digitizedfor electronic storage. These operations may occur independently of eachother, in series or simultaneously. Since the operations areindependent, they will be described separately, assuming for thepurposes of this example that film 101 is digitized first. However,there is no loss of generality and the same operations may be performedin any sequence noted above.

Film 101 is digitized 105 simultaneous with reading and decoding oftimecodes 107, which have been previously recorded in the margin of thefilm, for example. Decoded timecodes from film 101 are then used toestablish a reference timebase 109. The reference timebase is used toaccomplish synchronization as discussed below.

Since the timecodes recorded on film 101 represent absolute time atwhich each frame was recorded, there will ordinarily be time gapsbetween the last timecode of one take and the first timecode of asubsequent take. The source material is divided into clips at each suchdiscontinuity in the timecode. The reference timebase for each clip isessentially periodic timing information stored with the digitized clipand beginning at time 0 at the beginning of the clip. The informationultimately stored with each clip includes the digitized source, thedecoded timecodes and the reference time base. The reference time baseacts essentially as a frame or footage counter for locating a positionwithin the clip. In the case of film 101, these three types ofinformation are stored as picture information 111.

By a similar process, audio tape 103 is digitized 113 and stored 115.Like film 101, audio tape 103 also includes timecodes recorded therein.The timecodes are decoded 117 and used to establish a reference timebase109 to be stored along with the digitized audio information 115.

The beginning of clip of picture information and the clip of audioinformation is identified by a common starting timecode, thusestablishing time “O” of the reference time base. If film 101 and audiotape 103 were originally recorded simultaneously, corresponding storedpictures and stored audio are now synchronized by virtue of havingcommon reference timebases.

Sometimes the picture timecodes and audio timecodes recorded with eachsource to be synchornized are not common. For example, if one timecodegenerator was used for picture, and another timecode generator whoseclock was set differently was used for audio, then the recordedtimecodes are not in common. If multiple sources do not have commontimecodes, then an offset may be applied to bring them into alignment,if desired. That is, time “O” of the reference timebase for picture maybe one timecode, while that for audio may be a different timecode.However, once the timecodes for time “O” of the reference time base havebeen established, the common reference time base maintainssynchronization between the digitized sources.

A system suitable for performing synchronization as described above isnow described in connection with FIG. 2. This system is given by way ofexample only, and the principles of the present invention may beembodied in other hardware implementations.

The embodiment now described is based on modern computer technology. Forexample, some elements of this system may be incorporated within acommonly available personal computer system.

The system is controlled by a central processing unit (CPU) 201. CPU 201communicates with other elements of the system via address bus 203 anddata bus 205 in a conventional manner. Information including softwareprogram steps and other data which facilitate performance of the controlfunction by CPU 201 may be stored in read only memory (ROM) 207. Thesoftware program steps and data stored in ROM 207 are not readilyalterable. Additional program steps and data may be stored in some formof mass storage device 209, for example a hard disk drive or tape drive.In order to use program steps or data stored in ROM 207 or mass storagedevice 209, CPU 201 may transfer such data to a random access memory(RAM) 211. Finally the system may be provided with one or more user I/Odevices 213. These devices may include, but are not limited tokeyboards, pointing devices, monitors and printers. Taken together, theabove described elements of the system may constitute a personalcomputer system 215 including one or more add on user I/O devices 213.

To a system as described above, one or more media digitizers 217 andcorresponding timecode decoders 219 are added. A digitizer appropriateto each type of media to be used with the system is required, as well asa timecode decoder appropriate for use with any included media digitizerand suitable for decoding any expected timecode format. Digitizers 217which may be used in this embodiment include video cards manufactured byTRUEVISION and DIGIDESIGN. The Avid Film Composer employs such videocards, specially modified to provide additional functionality requiredby that product. Timecode decoders such as those manufactured by AATONor ARRIFLEX and commonly in use are suitable for application as timecodedecoders 219.

Once synchronization of digitized sources has been achieved as describedabove in connection with FIG. 1, certain novel playback techniques maybe used in accordance with another aspect of the present invention. Afirst such novel technique is now described in connection with FIG. 3.

When a director desires to periodically check the quality of work thenbeing filmed, he may rely on viewing those rolls of film shot in theprevious day, commonly known as “dailies.” Dailies are quickly processedand do not include integral audio information. Thus, the audiocorresponding to a roll of dailies must be synchronized with the roll,while it is being viewed, so as to provide a useful display of the day'swork.

In accordance with this aspect of the present invention, a personalcomputer 215, such as described above in connection with FIG. 2 is used.Mass storage device 209 holds the digitized audio informationcorresponding to the dailies to be shown, including the originaltimecodes and the information representing the reference timebase. Thecontrol function represented by certain computer program steps beingexecuted by CPU (FIG. 2, 201) give computer 215 the function of dailiessynchronizer 301. When the director views dailies, he views an image 303projected using raw, uncut film 305. As noted above, film 305 does notinclude the audio tracks, which have been recorded separately. It isthis audio which is held in mass storage device 209. The projector 307by which film 305 is shown may be fitted with a timecode reader 309.

Timecodes read off film 305 are decoded and fed into dailies player 301.The decoded timecodes are then converted into the reference timebase andused to retrieve corresponding audio information from mass storagedevice 209. Such audio information may then be reproduced by audioreproduction equipment 311, synchronous with the image 303 projected byprojector 307. Thus, the audio and picture synchronization whichpreviously had to be performed by hand prior to projecting film 305 maybe accomplished automatically.

In an alternate embodiment, timecode reader 309 may be replaced with abi-phase generator and a bi-phase-to-timecode converter, such as thatmanufactured by LYNX. This alternate arrangement produces timecodessynchronous with the running projector and beginning with any arbitrarytimecode an operator may set.

Another novel aspect of synchronization in accordance with the presentinvention is illustrated in connection with a negative cutter, as shownin FIG. 4.

Media 401, such as film or audio tape is threaded into sync block 403.Sync block 403 include a KEYKODE reader 405. KEYKODE readers arecommonly available from such manufacturers as EVERTZ, CINEMA PRODUCTS,AATON, ARRIFLEX and SKOTEL. KEYKODES are a reference code indicatingframe numbers on film media 401. However, KEYKODES are recorded on thefilm 401 only about once each ½ foot. Given an initial KEYKODE for aparticular piece of film 401, which is currently threaded into syncblock 403, a digital playback system 407 may be made to receive acontinuous indication of the position of media 401 within sync block 403by means of shaft encoder 408.

Another input to digital playback system 407 is a digital work print409. Digital workprint 409 is preferably an optical disk containing theresult of producing a composition using the method and apparatusdescribed in the above-referenced patent application Ser. No. 07/920,260The data contained on the digital workprint 409 includes the digitizedpicture and audio information, the original time codes associated witheach picture clip and audio clip, the reference time base informationand various sequencing information as described in the above-referencedpatent application with respect to a complete composition. By using theinformation described, digital playback system 407 can be used toindicate on a monitor 411, or by other means such as sounding a beeptone, when the media 401 threaded in sync block 403 is located at aposition where a cut is to be made. This is possible because the startand end of each clip on digital workprint 409 are clearly indicated bythe timecodes originally associated with that clip, and those timecodesmay be readily converted to KEYKODES, knowing the time between framesand the first KEYKODE on a particular piece of media. Thus, the editorof a composition need not produce a traditional paper cutlist, which maybe prone to transcription errors. Rather, the digital work print servesthe function of informing the negative cutter the precise location ofcuts to be made.

The present invention has now been described in connection with specificembodiments thereof. The foregoing description should suggest to thoseskilled in the art modifications and extensions of those embodiments,which are contemplated as falling within the scope of the presentinvention. The present invention is not intended to be limited by theforegoing description, but is limited only by the scope of the appendedclaims.

What is claimed is:
 1. A method for synchronizing multiple streams ofdigital media data, including a first stream of digital media dataincluding a first stream of timecodes and a second stream of digitalmedia data including a second stream of timecodes, the first stream oftimecodes having a first time resolution and the second stream oftimecodes having a second time resolution, the method comprising:determining a first relationship between a reference time base,beginning at a reference time zero and including periodic timinginformation, and one of the first and second streams of timecodes, thereference time base having a third time resolution; determining a secondrelationship between another of the first and second streams oftimecodes with the reference time base; and storing a representation ofthe first and second streams of digital media data as concurrentsynchronized streams using the reference time base according to thedetermined first and second relationships.
 2. The method of claim 1,wherein determining the first relationship further comprises: settingreference time zero to be coincident with a first timecode of one of thefirst stream of timecodes and the second stream of timecodes.
 3. Themethod of claim 1, wherein over a range of time of coincidence of thefirst stream of digital media data and the second stream of digitalmedia data, the first stream of timecodes differ from the second streamof timecodes and wherein determining the first relationship furthercomprises: setting the reference time zero to be coincident with a firsttimecode of the second stream of timecodes.
 4. An apparatus forsynchronizing digital audio data including audio timecodes with a filmsource, comprising: means for receiving time information from the filmsource representing a stream of picture timecodes; means for specifyinga reference time base beginning at a reference time zero and includingperiodic timing information synchronous with the picture timecodes;means for creating an association of the reference time zero with afirst audio timecode of the audio timecodes; and means for storing arepresentation of the film source and the audio as concurrentsyncrhonized streams according to the association created between thereference time base and the audio timecodes.
 5. Apparatus forsynchronizing multiple streams of digital media data including a firststream of digital media data including a first stream of timecodes and asecond stream of digital media data including a second stream oftimecodes, the first stream of timecodes having a first time resolutionand the second stream of timecodes having a second time resolution, theapparatus comprising: means for determining a first relationship betweena reference time base, beginning at a reference time zero and includingperiodic timing information, and one of the first and second streams oftimecodes, the reference time base having a third time resolution; meansfor determining a second relationship between another of the first andsecond stream of timecodes with the reference time base; and means forstoring a representation of the first and second streams of digitalmedia data as concurrent synchronized streams using the reference timebase according to the determined first and second relationships.
 6. Theapparatus of claim 5, wherein the means for determining the firstrelationship further comprises: means for establishing reference timezero as coincident with a first timecode of one of the first stream oftimecodes and the second stream of timecodes.
 7. The apparatus of claim5, wherein over a range of time of coincidence of the first stream ofdigital media data and the second stream of digital media data, thefirst stream of timecodes differ from the second stream of timecodes andwherein the means for determining the first relationship furthercomprises: means for establishing the reference time zero as coincidentwith a first timecode of the second stream of timecodes.
 8. A method forsynchronizing digital audio data including audio timecodes with a filmsource, comprising: receiving time information from the film sourcerepresenting a stream of picture timecodes; specifying a reference timebase beginning at a reference time zero and including periodic timinginformation synchronous with the picture timecodes; creating anassociation of the reference time zero with a first audio timecode ofthe audio timecodes; and storing a representation of the film source andthe audio as concurrent syncrhonized streams according to theassociation created between the reference time base and the audiotimecodes.
 9. A method for automatically synchronizing a plurality ofindependently captured streams of digital media data to each other,wherein each stream has corresponding time codes, comprising:determining a first relationship between a start time code of a firststream of the plurality of streams of digital media data and a referencetime zero of a reference time base; determining a second relationshipbetween a start time code of a second stream of the plurality of streamsof digital media data and the reference time zero of a reference timebase; and creating a data structure describing the first and secondstreams of digital media data as a collection of concurrent componentshaving a common start time of the reference time zero, in accordancewith the first relationship and the second relationship, whereby thefirst and second streams of digital media data are synchronized witheach other.
 10. The method of claim 9 wherein a first component in thedata structure describes the first stream and references the start timecode of the first stream and a second component in the data structuredescribes the second stream and references the start time code of thesecond stream.
 11. The method of claim 9, wherein the first stream andthe second stream have different frame rates.
 12. The method of claim11, wherein the time codes of the first stream and the time codes of thesecond stream are coincident in absolute time.
 13. The method of claim11, wherein the time codes of the first stream and the time codes of thesecond stream are not coincident in absolute time, whereby the starttime code for the first stream is different from the start time code ofthe second stream.
 14. The method of claim 9, wherein the first streamand the second stream have the same frame rate.
 15. The method of claim14, wherein the time codes of the first stream and the time codes of thesecond stream are coincident in absolute time.
 16. The method of claim14, wherein the time codes of the first stream and the time codes of thesecond stream are not coincident in absolute time, whereby the starttime code for the first stream is different from the start time code ofthe second stream.
 17. The method of claim 9, wherein the time codes ofthe first stream and the time codes of the second stream are coincidentin absolute time.
 18. The method of claim 9, wherein the time codes ofthe first stream and the time codes of the second stream are notcoincident in absolute time, whereby the start time code for the firststream is different from the start time code of the second stream. 19.The method of claim 9, wherein the first stream is video and the secondstream is audio.
 20. An apparatus for automatically synchronizing aplurality of independently captured streams of digital media data toeach other, wherein each stream has corresponding time codes,comprising: means for determining a first relationship between a starttime code of a first stream of the plurality of streams of digital mediadata and a reference time zero of a reference time base; means fordetermining a second relationship between a start time code of a secondstream of the plurality of streams of digital media data and thereference time zero of a reference time base; and means for storing adata structure describing the first and second streams of digital mediadata as a collection of concurrent components having a common start timeof the reference time zero, in accordance with the first relationshipand the second relationship, whereby the first and second streams ofdigital media data are synchronized with each other.
 21. The apparatusof claim 20, wherein a first component in the data structure describesthe first stream and references the start time code of the first streamand a second component in the data structure describes the second streamand references the start time code of the second stream.
 22. Theapparatus of claim 20, wherein the first stream and the second streamhave different frame rates.
 23. The apparatus of claim 22, wherein thetime codes of the first stream and the time codes of the second streamare coincident in absolute time.
 24. The apparatus of claim 22, whereinthe time codes of the first stream and the time codes of the secondstream are not coincident in absolute time, whereby the start time codefor the first stream is different from the start time code of the secondstream.
 25. The apparatus of claim 20, wherein the first stream and thesecond stream have the same frame rate.
 26. The apparatus of claim 25,wherein the time codes of the first stream and the time codes of thesecond stream are coincident in absolute time.
 27. The apparatus ofclaim 25, wherein the time codes of the first stream and the time codesof the second stream are not coincident in absolute time, whereby thestart time code for the first stream is different from the start timecode of the second stream.
 28. The apparatus of claim 20, wherein thetime codes of the first stream and the time codes of the second streamare coincident in absolute time.
 29. The apparatus of claim 20, whereinthe time codes of the first stream and the time codes of the secondstream are not coincident in absolute time, whereby the start time codefor the first stream is different from the start time code of the secondstream.
 30. The apparatus of claim 20, wherein the first stream is videoand the second stream is audio.